-
Notifications
You must be signed in to change notification settings - Fork 695
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Set BWA and bwamem2 index memory dynamically #6628
base: master
Are you sure you want to change the base?
Conversation
This one doesn't fail, but it's good practice.
9d9d274
to
f10c3f8
Compare
I like it 👍 it’ll also play nice with the new resourceLimits directive |
I was talking to @drpatelh about this earlier this week. Sounds good. Very neat if it scales in such a linear way. Should we add a baseline of additional memory? |
Closes nf-core/sarek#1377 |
What can we do for bwa mem and bwamem2 mem? |
What do you mean? |
Is it right to have these settings hardcoded in the module ? How does it interact with pipeline-level config file doing
which one takes precedence ? |
AFAIK the pipeline config takes precedence over the config hardcoded in the module. |
I was more worried that 28 GB / Gbp is still too high in my view. I use 24 GB / Gbp in my pipelines and wouldn't want nf-core to force me to waste RAM ;) I wasn't worried of |
FYI, I've just checked our LSF logs and there's been zero memory failures over the 1,698 Regardless of the scaling factor you use, I'd still keep |
Kept having bwamem2 index tasks that ran forever and failed.
Updated bwamem2 to use 28.B of memory per byte of fasta. Issue for reference: bwa-mem2/bwa-mem2#9
Also tracked down the required memory for bwa index while I was at it. Doesn't seem to fail because most of the genome requirements are under the necessary memory.
Not the first place where people have run into this #6628